System and methods for planning and performing three-dimensional holographic interventional procedures with three-dimensional tomographic and live imaging

ABSTRACT

A method and a system for image-guided intervention such as a percutaneous treatment or diagnosis of a patient may include at least one of a pre-registration method and a re-registration method. The pre-registration method is configured to permit for an efficient virtual representation of a planned trajectory to target tissue during the intervention, for example, as a holographic light ray shown through an augmented reality system. In turn, this allows the operator to align a physical instrument such as a medical probe for the intervention. The re-registration method is configured to adjust for inaccuracy in the virtual representation generated by the pre-registration method, as determined by live imaging of the patient during the intervention. The re-registration method may employ the use of intersectional contour lines to define the target tissue as viewed through the augmented reality system, which permits for an unobstructed view of the target tissue for the intervention.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Ser.No. 62/945,983, filed on Dec. 10, 2019. The entire disclosure of theabove application is hereby incorporated herein by reference.

FIELD

The present disclosure relates to holographic augmented realityapplications and, more particularly, medical applications employingholographic augmented reality.

INTRODUCTION

This section provides background information related to the presentdisclosure which is not necessarily prior art.

Holographic augmented reality technology is finding more widespread usein healthcare applications to improve medical procedures, clinicaloutcomes, and long-term patient care. These augmented realitytechnologies are also useful for enhancing the real environments in thepatient care setting, for example, with content-specific information toimprove patient outcomes. For example, a practitioner can viewadditional information in the same field of view while performing amedical procedure, where the practitioner does not have to change theirgaze, which may otherwise slow down or reduce the efficiency of theprocedure.

More specifically, image-guided intervention during medical proceduressuch as minimally invasive surgical (MIS) procedures is inherently ahighly three-dimensional (3D) task. For example, percutaneous ablationof solid tumors requires accurate 3D positioning of one or more thermalprobes while the tumor is undergoing respiratory motion or ventilation.However, flat or two-dimensional (2D) display monitors are often used todisplay the image-guidance data in the standard of care for MISprocedures. 3D holographic guidance and navigation, using one or moreimaging modalities, can provide improved depth perception and spatialunderstanding during image-guided intervention, but the spatialregistration of 3D guidance and navigation holograms must be more 1)accurate and 2) easy to use (i.e., operator workflow must be effectiveand efficient) for these methods to be adopted relative to more invasivesurgery.

As described in U.S. Patent Application Publication No. 2018/0303563 toWest et al., it is known to use 3D holographic image-guidance to trackan interventional device during a percutaneous procedure. The 3Dholographic image-guidance can be provided by a head-mounted device bytransforming tracking data and body image data to a common coordinatesystem and creating a holographic display relative to a body of apatient to track the interventional device during the non-vascularpercutaneous procedure. The holographic display can also includegraphics to provide guidance for the physical interventional device asit travels through the anatomy of the patient.

There is a continuing need for a holographic system and method thatenables pre-registration for 3D holographic guidance and navigationleading to less steps for the operator and more effective workflow,thereby facilitating increased adoption of the system and method.Desirably, the system and method enable intra-procedural re-registrationof 3D holograms derived from 3D tomographic data with live imaging toimprove accuracy and safety of the probe placement during the MISprocedure.

SUMMARY

In concordance with the instant disclosure, a holographic system andmethod that enables pre-registration for 3D holographic guidance andnavigation leading to less steps for the operator and more effectiveworkflow, thereby facilitating increased adoption of the system andmethod, and which enables intra-procedural re-registration of 3Dholograms derived from 3D tomographic data with live imaging to improveaccuracy and safety of the probe placement during the MIS procedure, hasbeen surprisingly discovered.

In one embodiment, a system and method of image-guided intervention fora patient involving pre-registration includes provision of aninstallation of an augmented reality system such as a MicrosoftHoloLens® at a first image acquisition apparatus. The augmented realitysystem is in communication with a computer having a processor and amemory, and the computer may be integrated with augmented realitysystem. The augmented reality system may initially be disposed at apredetermined storage position. The augmented reality system has anaugmented reality system coordinate system and the first imageacquisition apparatus has a first image acquisition apparatus coordinatesystem. The first image acquisition apparatus further has a patienttable for receiving a patient and an imager for acquiring a firstholographic image dataset from the patient while on the patient table.

The installation further includes a step of placing a first opticalimage target at a predetermined location on the imager. First opticalimage target coordinates are then acquired from the first optical imagetarget with the augmented reality system. A pre-registrationtransformation of the first image acquisition apparatus coordinatesystem into the augmented reality system coordinate system is thendetermined using the first optical image target coordinates. Thepre-registration transformation is stored in the memory of the computer,whereby a pre-registration of the first image acquisition apparatus withthe augmented reality system is performed during the installation.

Following the installation of the augmented reality system and thepre-registration of the first image acquisition apparatus with theaugmented reality system, the method further includes a step of applyingthe pre-registration transformation to the first holographic imagedataset. This transforms the first holographic image dataset from thefirst image acquisition apparatus coordinate system to the augmentedreality system coordinate system. Advantageously, this instance of thepre-registration can be used repeatably and improves the efficiency ofthe operator performing the image-guided intervention by eliminatingcertain registration procedures that would otherwise need to beperformed while the patient is on the table for the image-guidedintervention.

In another embodiment, a system and a method of image-guidedintervention for a patient involving re-registration includes provisionof an installation of an augmented reality system such as a MicrosoftHoloLens® at a first image acquisition apparatus. The augmented realitysystem is in communication with a computer with a processor and amemory. The augmented reality system may initially be disposed at apredetermined storage position. The augmented reality system has anaugmented reality system coordinate system and the first imageacquisition apparatus has a first image acquisition apparatus coordinatesystem. An initial transformation, such as, but not limited to, apre-registration transformation of the first image acquisition apparatuscoordinate system into the augmented reality system coordinate system,is stored in the memory of the computer. The first image acquisitionapparatus further has a patient table for receiving a patient and animager for acquiring a first holographic image dataset from the patientwhile on the patient table.

The method further includes a step of acquiring with one of the firstimage acquisition apparatus and a second image acquisition apparatus areal-time holographic image dataset of the patient during theimage-guided intervention. The real-time holographic image dataset isthen compared with the patient or the first holographic image dataset.The first holographic image dataset is then adjusted, either manually bythe operator or automatically according to predetermined rules, to alignthe first holographic image dataset with the real-time holographic imagedataset from the patient. This adjustment thereby provides are-registration transformation. The re-registration transformation maybe stored in the memory of the computer.

In a further embodiment, a system and a method of image-guidedintervention for a patient involving an intersectional contour isprovided. The intersectional contour technique may be employed as partof a re-registration system and a method as described, or may be used inother contexts relative to the image-guided intervention, as desired.The method includes a step of acquiring with a first image acquisitionapparatus a first holographic image dataset of the patient, andacquiring with a second image acquisition apparatus a second holographicimage dataset of the patient during the image-guided intervention. Thefirst holographic image dataset is then compared with the secondholographic image dataset, for example, by overlaying the firstholographic image dataset with the second holographic image dataset. Theintersectional contour of the first holographic image dataset on thesecond holographic image dataset is then determined. A portion of thefirst holographic image dataset that is not the intersectional contourmay then be removed from a view of an operator of the augmented realitysystem. Only the intersectional contour of the first holographic imagedataset is thereby shown overlaid on the second holographic imagedataset via the augmented reality system.

Further areas of applicability will become apparent from the descriptionprovided herein. It should be understood that the description andspecific examples are intended for purposes of illustration only and arenot intended to limit the scope of the present disclosure.

DRAWINGS

The drawings described herein are for illustrative purposes only ofselected embodiments and not all possible implementations and are notintended to limit the scope of the present disclosure.

FIG. 1 is a flow diagram illustrating a method for performing apre-registration associated with an image-guided intervention for apatient, according to one embodiment of the disclosure;

FIG. 2 is a schematic diagram of a system according to one embodiment ofthe disclosure, the system adapted for use in the pre-registrationmethod of FIG. 1;

FIG. 3 is a perspective view of an imaging system according to oneembodiment of the disclosure and adapted for use with a holographicsystem and method, the imaging system shown as a multirow detectorcomputerized tomography (MDCT) scanner with associated table;

FIG. 4A is a perspective view of an imaging system according to anotherembodiment of the disclosure and adapted for use with a holographicsystem and a method, the imaging system shown as a C-arm angiographyfluoroscopy unit (Cone beam CT or CBCT) with the X-ray source anddetectors mounted on a C-arm apparatus adjacent an associated table;

FIG. 4B is a perspective view of the imaging system shown in FIG. 4A,but with the C-arm angiography fluoroscopy unit moved away from thetable and a holographic light ray remaining at the table for theoperator to use in trajectory planning without interference from theC-arm angiography fluoroscopy unit;

FIG. 5 is a schematic diagram showing use of the multirow detectorcomputerized tomography (CT) scanner shown in FIG. 3 with the method andthe system shown in FIGS. 1 and 2;

FIG. 6 is a flowchart illustrating a method or workflow forintra-operative pre-registration of the imaging system shown in FIG. 5to a coordinate system of a head mounted display of the holographicsystem;

FIG. 7 is a schematic diagram showing use of the C-arm angiographyfluoroscopy unit shown in FIGS. 4A and 4B with the method and systemshown in FIGS. 1 and 2;

FIG. 8 is a flowchart illustrating a method or workflow forintra-operative pre-registration of the imaging system shown in FIGS. 4Aand 4B to a coordinate system of a head mounted display of theholographic system;

FIG. 9 is a flow diagram illustrating a method for performing are-registration associated with an image-guided intervention for apatient, according to another embodiment of the disclosure;

FIG. 10 is a schematic diagram of a system according to anotherembodiment of the disclosure, the system adapted for use in there-registration method of FIG. 9;

FIG. 11 is a schematic diagram showing a modeled camera geometry forlive imaging used for re-registration of a treatment plan including 3Danatomical structures and associated instrument trajectories to the headmounted display coordinates, according to various embodiments of thedisclosure;

FIG. 12 is a schematic diagram showing a re-registration method with alive imaged key structure having a black solid contour, and anintersection between the 3D key structure of the treatment plan and thelive image field-of-view (planar in the case) shown as a dashed contourline, where after re-registration the contours are substantiallycongruent and the planned trajectory accurately aligned;

FIG. 13 is a flow diagram illustrating a method for generation of anintersectional contour for a re-registration associated with animage-guided intervention for a patient, according to a furtherembodiment of the disclosure;

FIG. 14 is a table depicting an algorithm for the generation of theintersectional contour of FIG. 13, according to the further embodimentof the disclosure;

FIG. 15 is schematic view of an exemplary image-guided intervention forthe patient involving the intersectional contour of FIG. 13, theexemplary image guided intervention including a real-timetwo-dimensional sector image of the patient compared with athree-dimensional holographic image of the patient (left), andre-registration of the three-dimensional holographic image of thepatient with the two-dimensional sector image of the patient (right);

FIG. 16 is a top perspective view of the exemplary image-guidedintervention shown in FIG. 15, from a point of view of an operator ofthe augmented reality system, and further showing an ultrasound scanningof a portion of a patient with an ultrasound probe and the ultrasoundprobe being pressed against the patient, and an offset first opticalimage target for use with the augmented reality system;

FIG. 17 is a top perspective view of the image-guided intervention shownin FIG. 16, and further illustrating a three-dimensional holographicimage of the patient from a MDCT scan of the patient, thethree-dimensional holographic image showing target tissue for theintervention;

FIG. 18 is a top perspective view of the image-guided intervention shownin FIG. 17, and further illustrating a real-time two-dimensional sectorimage from the ultrasound scan overlaid with the three-dimensionalholographic image of the patient from the MDCT scan;

FIG. 19 is a top perspective view of the image-guided intervention shownin FIG. 18, and further illustrating a generation of an intersectionalcontour of the target tissue on the ultrasound plane; and

FIG. 20 is a top perspective view of the image-guided intervention shownin FIG. 19, and further illustrating a change in shape of theintersectional contour on the real-time two-dimensional sector imagewith movement of the ultrasound probe.

DETAILED DESCRIPTION

The following description of technology is merely exemplary in nature ofthe subject matter, manufacture and use of one or more inventions, andis not intended to limit the scope, application, or uses of any specificinvention claimed in this application or in such other applications asmay be filed claiming priority to this application, or patents issuingtherefrom. Regarding methods disclosed, the order of the steps presentedis exemplary in nature unless otherwise disclosed, and thus, the orderof the steps can be different in various embodiments, including wherecertain steps can be simultaneously performed.

I. Definitions

Unless otherwise defined, all technical terms used herein have the samemeaning as commonly understood by one of ordinary skill in the art towhich the present disclosure pertains.

As used herein, the terms “a” and “an” indicate “at least one” of theitem is present; a plurality of such items may be present, whenpossible. Except where otherwise expressly indicated, all numericalquantities in this description are to be understood as modified by theword “about” and all geometric and spatial descriptors are to beunderstood as modified by the word “substantially” in describing thebroadest scope of the technology. “About” when applied to numericalvalues indicates that the calculation or the measurement allows someslight imprecision in the value (with some approach to exactness in thevalue; approximately or reasonably close to the value; nearly). If, forsome reason, the imprecision provided by “about” and/or “substantially”is not otherwise understood in the art with this ordinary meaning, then“about” and/or “substantially” as used herein indicates at leastvariations that may arise from ordinary methods of measuring or usingsuch parameters.

Although the open-ended term “comprising,” as a synonym ofnon-restrictive terms such as including, containing, or having, is usedherein to describe and claim embodiments of the present technology,embodiments may alternatively be described using more limiting termssuch as “consisting of” or “consisting essentially of.” Thus, for anygiven embodiment reciting materials, components, or process steps, thepresent technology also specifically includes embodiments consisting of,or consisting essentially of, such materials, components, or processsteps excluding additional materials, components or processes (forconsisting of) and excluding additional materials, components orprocesses affecting the significant properties of the embodiment (forconsisting essentially of), even though such additional materials,components or processes are not explicitly recited in this application.For example, recitation of a process reciting elements A, B and Cspecifically envisions embodiments consisting of, and consistingessentially of, A, B and C, excluding an element D that may be recitedin the art, even though element D is not explicitly described as beingexcluded herein.

As referred to herein, disclosures of ranges are, unless specifiedotherwise, inclusive of endpoints and include all distinct values andfurther divided ranges within the entire range. Thus, for example, arange of “from A to B” or “from about A to about B” is inclusive of Aand of B. Disclosure of values and ranges of values for specificparameters (such as amounts, weight percentages, etc.) are not exclusiveof other values and ranges of values useful herein. It is envisionedthat two or more specific exemplified values for a given parameter maydefine endpoints for a range of values that may be claimed for theparameter. For example, if Parameter X is exemplified herein to havevalue A and also exemplified to have value Z, it is envisioned thatParameter X may have a range of values from about A to about Z.Similarly, it is envisioned that disclosure of two or more ranges ofvalues for a parameter (whether such ranges are nested, overlapping ordistinct) subsume all possible combination of ranges for the value thatmight be claimed using endpoints of the disclosed ranges. For example,if Parameter X is exemplified herein to have values in the range of1-10, or 2-9, or 3-8, it is also envisioned that Parameter X may haveother ranges of values including 1-9,1-8,1-3,1-2,2-10,2-8,2-3,3-10,3-9,and so on.

When an element or layer is referred to as being “on,” “engaged to,”“connected to,” or “coupled to” another element or layer, it may bedirectly on, engaged, connected or coupled to the other element orlayer, or intervening elements or layers may be present. In contrast,when an element is referred to as being “directly on,” “directly engagedto,” “directly connected to” or “directly coupled to” another element orlayer, there may be no intervening elements or layers present. Otherwords used to describe the relationship between elements should beinterpreted in a like fashion (e.g., “between” versus “directlybetween,” “adjacent” versus “directly adjacent,” etc.). As used herein,the term “and/or” includes any and all combinations of one or more ofthe associated listed items.

Although the terms first, second, third, etc. may be used herein todescribe various elements, components, regions, layers and/or sections,these elements, components, regions, layers and/or sections should notbe limited by these terms. These terms may be only used to distinguishone element, component, region, layer or section from another region,layer or section. Terms such as “first,” “second,” and other numericalterms when used herein do not imply a sequence or order unless clearlyindicated by the context. Thus, a first element, component, region,layer or section discussed below could be termed a second element,component, region, layer or section without departing from the teachingsof the example embodiments.

Spatially relative terms, such as “inner,” “outer,” “beneath,” “below,”“lower,” “above,” “upper,” and the like, may be used herein for ease ofdescription to describe one element or feature's relationship to anotherelement(s) or feature(s) as illustrated in the figures. Spatiallyrelative terms may be intended to encompass different orientations ofthe device in use or operation in addition to the orientation depictedin the figures. For example, if the device in the figures is turnedover, elements described as “below” or “beneath” other elements orfeatures would then be oriented “above” the other elements or features.Thus, the example term “below” can encompass both an orientation ofabove and below. The device may be otherwise oriented (rotated 90degrees or at other orientations) and the spatially relative descriptorsused herein interpreted accordingly.

As used herein, the term “percutaneous” refers to something that ismade, done, or effected through the skin.

As used herein, the term “percutaneous medical procedure” refers toaccessing the internal organs or tissues via needle-puncture of theskin, rather than by using an open approach where the internal organs ortissues are exposed (typically with a scalpel).

As used herein, the term “non-vascular” when used with “percutaneousmedical procedure” refers to a medical procedure performed on anyportion of the subject's body distinct from the vasculature that isaccessed percutaneously. Examples of percutaneous medical procedures caninclude a biopsy, a tissue ablation, a cryotherapy procedure, abrachytherapy procedure, an endovascular procedure, a drainage procedurean orthopedic procedure, a pain management procedure, a vertebroplastyprocedure, a pedicle/screw placement procedure, a guidewire-placementprocedure, a SI-Joint fixation procedure, a training procedure, or thelike.

As used herein, the term “interventional device” refers to a medicalinstrument used during the non-vascular percutaneous medical procedure.

As used herein, the term “tracking system” refers to something used toobserve one or more objects undergoing motion and supply a timelyordered sequence of tracking data (e.g., location data, orientationdata, or the like) in a tracking coordinate system for furtherprocessing. As an example, the tracking system can be an electromagnetictracking system that can observe an interventional device equipped witha sensor-coil as the interventional device moves through a patient'sbody.

As used herein, the term “tracking data” refers to information recordedby the tracking system related to an observation of one or more objectsundergoing motion.

As used herein, the term “tracking coordinate system” refers to a 3DCartesian coordinate system that uses one or more numbers to determinethe position of points or other geometric elements unique to theparticular tracking system. For example, the tracking coordinate systemcan be rotated, scaled, or the like, from a standard 3D Cartesiancoordinate system.

As used herein, the term “head-mounted device” or “headset” or “HIVID”refers to a display device, configured to be worn on the head, that hasone or more display optics (including lenses) in front of one or moreeyes. These terms may be referred to even more generally by the term“augmented reality system.” In some instances, the head-mounted devicecan also include a non-transitory memory and a processing unit. Anexample of a suitable head-mounted device is a Microsoft HoloLens®.

As used herein, the term “imaging system” or “image acquisitionapparatus” or the like refers to technology that creates a visualrepresentation of the interior of a patient's body. For example, theimaging system can be a computed tomography (CT) system, a fluoroscopysystem, a magnetic resonance imaging (MRI) system, an ultrasound (US)system, or the like.

As used herein, the terms “coordinate system” or “augmented realtysystem coordinate system” refers to a 3D Cartesian coordinate systemthat uses one or more numbers to determine the position of points orother geometric elements unique to the particular augmented realitysystem or image acquisition apparatus to which it pertains. For example,the headset coordinate system can be rotated, scaled, or the like, froma standard 3D Cartesian coordinate system.

As used herein, the terms “image data” or “image dataset” refers toinformation recorded in 3D by the imaging system related to anobservation of the interior of the patient's body. For example, the“image data” or “image dataset” can include processed two-dimensional orthree-dimensional images or models such as tomographic images, e.g.,represented by data formatted according to the Digital Imaging andCommunications in Medicine (DICOM) standard or other relevant imagingstandards.

As used herein, the terms “imaging coordinate system” or “imageacquisition apparatus coordinate system” refers to a 3D Cartesiancoordinate system that uses one or more numbers to determine theposition of points or other geometric elements unique to the particularimaging system. For example, the imaging coordinate system can berotated, scaled, or the like, from a standard 3D Cartesian coordinatesystem.

As used herein, the terms “hologram”, “holographic,” “holographicprojection”, or “holographic representation” refer to acomputer-generated image projected to a lens of a headset. Generally, ahologram can be generated synthetically (in an augmented reality (AR))and is not related to physical reality.

As used herein, the term “physical” refers to something real. Somethingthat is physical is not holographic (or not computer-generated).

As used herein, the term “two-dimensional” or “2D” refers to somethingrepresented in two physical dimensions.

As used herein, the term “three-dimensional” or “3D” refers to somethingrepresented in three physical dimensions. An element that is “4D” (e.g.,3D plus a time and/or motion dimension) would be encompassed by thedefinition of three-dimensional or 3D.

As used herein, the term “integrated” can refer to two things beinglinked or coordinated. For example, a coil-sensor can be integrated withan interventional device.

As used herein, the term “degrees-of-freedom” or “DOF” refers to anumber of independently variable factors. For example, a tracking systemcan have six degrees-of-freedom (or 6DOF), a 3D point and 3 dimensionsof rotation.

As used herein, the term “real-time” refers to the actual time duringwhich a process or event occurs. In other words, a real-time event isdone live (within milliseconds so that results are available immediatelyas feedback). For example, a real-time event can be represented within100 milliseconds of the event occurring.

As used herein, the terms “subject” and “patient” can be usedinterchangeably and refer to any vertebrate organism.

II. Pre-registration

FIGS. 1-2 illustrate a method 100 and a system 200 of image-guidedintervention involving pre-registration, as described further herein.The method 100 includes a step 102 of providing the system 200, as shownin FIG. 2, for performing the image-guided intervention on a patient201, The system 200 includes an installation 202 of an augmented realitysystem 204 at a first image acquisition apparatus 206. The image-guidedintervention may be, as one non-limiting example, a percutaneous medicalprocedure.

As non-limiting examples, and as illustrated further in FIGS. 3 and 4,the first image acquisition apparatus 206 may be one of a multidetectorrow computerized tomography (MDCT) imager (shown in FIG. 3) and a C-armangio fluoroscopy imager (shown in FIGS. 4A and 4B). One of ordinaryskill in the art may also select other suitable types of imaging systemsfor the first image acquisition apparatus 206 within the scope of thepresent disclosure.

The augmented reality system 204 is in communication with a computer 208having a processor 209 and a memory 211. The memory 211 may includenon-transitory processor-executable instructions directing the augmentedreality system 204 to depict a first holographic image dataset adjacentto the patient 201. The first holographic image dataset may define athree-dimensional image volume, for example. The computer 208 may beconfigured to receive the surgical interventional plan from the firstimage acquisition apparatus 206.

In a particular embodiment, the augmented reality system 204 may includea stereoscopic head mounted display system such as the MicrosoftHoloLens®, with a tracking system (inertial measurement unit), anintegrated CPU and holographic processing unit, a camera, andholographic projection lenses. Other suitable types of the augmentedreality system 204 may also be selected by a skilled artisan within thescope of the present disclosure.

The augmented reality system 204 may be initially disposed at apredetermined storage position 210 relative to the first imagingacquisition apparatus 206. The augmented reality system 204 may have anaugmented reality system coordinate system 212, and the first imageacquisition apparatus 206 may have a first image acquisition apparatuscoordinate system 214. The augmented reality system coordinate system212 is different from the first image acquisition apparatus coordinatesystem 214.

In particular embodiments, the first image acquisition apparatuscoordinate system 214 and the augmented reality system coordinate system212 may be depicted through use of phantom lines with a delineatedcoordinate axis aligned with imagers, lasers, or phantom lines alignedusing imaging results at a fixed patient table position.

The computer 208 is also configured to transform the first imageacquisition apparatus coordinate system 214 to the augmented realitysystem coordinate system 212 for the first holographic image dataset,for example, according to the pre-registration transformation techniqueof the present disclosure.

The first image acquisition apparatus 206 further has a patient table216 for receiving the patient 201 and an imager 218 for acquiring afirst holographic image dataset from the patient 201 while on thepatient table 216.

With further reference to FIGS. 1 and 2, the method 100 further includesan installation step 101. The installation step 101 may include a step104 of placing a first optical image target 220 at a predeterminedlocation on the imager 218. In particular embodiments, where thepredetermined location for the first optical image target 220 is theimager 218, the image-guided intervention for the patient 201 mayfurther includes a step of aligning the imager 218 with the trajectoryfor the interventional instrument to be used in the image-guidedintervention for the patient 201.

In a step 106, first optical image target coordinates are then acquiredfrom the first optical image target 220 with the augmented realitysystem 204. The step 106 of acquiring the first optical image targetcoordinates from the first optical image target 220 with the augmentedreality system 204 may include moving the augmented reality system 204so that the first optical image target 220 is in a field-of-view of theaugmented reality system 204.

A pre-registration transformation is then determined in a step 108 usingthe first optical image target coordinates, with the pre-registrationtransformation being of the first image acquisition apparatus coordinatesystem 214 into the augmented reality system coordinate system 212. Thepre-registration transformation may involve rigid-body affine, as anon-limiting example.

In a step 110, the pre-registration transformation is stored in thememory 211 of the computer 208, whereby a pre-registration of the firstimage acquisition apparatus 206 with the augmented reality system 204 isperformed. It should be understood that the pre-registration may then bere-used for each subsequent patient, and does not need to be performedagain unless the system does not verify the accuracy or validity of thepre-registration.

Following the installation step 101 of the augmented reality system 204and the pre-registration of the first image acquisition apparatus 206with the augmented reality system 204, the method 100 further includes astep 112 of applying the pre-registration transformation to transformthe first holographic image dataset from the first image acquisitionapparatus coordinate system 214 to the augmented reality systemcoordinate system 212.

In particular, the step 112 of applying the pre-registrationtransformation may occur during the image-guided intervention, which inturn may include a step 114 of placing a second image target 222 on thepatient 201 while on the patient table 216. Second optical image targetcoordinates may then be acquired in a step 116 from the second imagetarget 222 with the augmented reality system 204. In a step 118, a firstholographic image dataset of a portion of the patient 201 on the patienttable 216 is then acquired.

With the first holographic image dataset acquired, the operator then, ina step 120, creates a surgical interventional plan using the first imageacquisition apparatus 206, for example, using hardware and software ofthe first image acquisition apparatus 206. The surgical interventionalplan including a delineation of target tissue in the first holographicimage dataset, for example, the identification of the target tissue in athree-dimensional holographic image or model derived from the firstholographic image dataset. It should be appreciated that the surgicalinterventional plan is likewise provided in the first image acquisitionapparatus coordinate system 214.

In a step 122, the surgical interventional plan is then transmitted fromthe first image acquisition apparatus 206 to the augmented realitysystem 204. Upon being transmitted in step 122, the surgicalinterventional plan in then transformed in a step 124 from a first imageacquisition apparatus coordinate system 214 of the first imageacquisition apparatus into the augmented reality system coordinatesystem 212 using the pre-registration transformation from theinstallation step 101. Subsequently, the augmented reality system 204may be used in a step 126 to generate a holographic light ray (shown inFIGS. 4A and 4B, for example) to show the operator a trajectory for theinterventional instrument according to the surgical interventional plan,for use in the image-guided intervention.

It should be appreciated that, in certain embodiments, such as in thecase of a multi-detector row CT scanner, the patient table 216 istranslatable or movable. In this case, the method 100 may furtherinclude at least one of the following steps: i) determining andregistering a position of the patient table 216 with the augmentedreality system 204; ii) placing an additional image target (not shown)at the patient table 216, and acquiring an additional image targetdataset from the additional image target with the augmented realitysystem 204, and determining a position of the patient table 216 by theaugmented reality system 204 based on the additional image targetdataset; and iii) transmitting a position of the patient table 216 froma patient table sensor (shown as “217” in FIG. 6) on the patient table216 to the augmented reality system 204. Other suitable means forproviding relevant information on the position of the patient table 216to the augmented reality system 204, for use in creating thepre-registration transformation, may also be employed within the scopeof the present disclosure.

Example Pre-Registration Techniques:

1. Summary:

With reference to FIGS. 3 and 5-6, and FIGS. 4 and 7-8, furtherexemplary embodiments associated with the pre-registration techniquesdescribed hereinabove are shown and explained. Like or related structurein FIGS. 3 and 5-6, in comparison to that shown in FIGS. 1-2, arereproduced in FIGS. 3 and 5-6 with a same reference number and a prime(') symbol for purpose of clarity. Like or related structure in FIGS.4A-4B and 7-8, in comparison to that shown in FIGS. 1-2, are reproducedin FIGS. 4A-4B and 7-8 with a same reference number and a prime (′)symbol for purpose of clarity.

In particular embodiments, as illustrated in FIGS. 3-8, aminimally-invasive surgical intervention may be planned in the system200′, 200″ where the first image acquisition apparatus 206′, 206″ istomographic imaging system such as a multi-detector row CT (MDCT, shownin FIGS. 3 and 5-6) or a C-arm angio fluoroscopy imager (shown in FIGS.4A-4B and 7-8), as described further herein. The system 200′, 200″ mayfurther include the augmented realty system 204′, 204″ in the form of ahead mounted display (HMD) such as a Microsoft HoloLens® including thecomputer 208′ , 208″ with the associated processor 209′, 209″ and thememory 211′, 211″. The system 200′, 200″ may also include thepredetermined storage position 210′, 210″ in the form of an HMD dockingstation or storage pod disposed adjacent to the first image acquisitionapparatus 206′, 206″, a table position sensor transmitter 217′ (in thecase of the MDCT) or an alert module for table movement 219″ (in thecase of the C-arm angio fluoroscopy imager), and the first optical imagetarget 220′, 220″ (only during the installation step 101′, 101″). Thepatient table sensor 217′ or the alert module 219″ may be configured fordetecting a position of the patient table 216′, 216″ and fortransmitting to the augmented reality system 204′, 204″ the position ofthe patient table 216′, 216″, in operation.

The system 200′, 200″ further may include a computer 224′, 224″associated with the MDCT or a C-arm angio fluoroscopy imager. Thecomputer 224′, 224″ may have a computer display 226′, 226″ and a cursorcontroller 228′, 228″ configured to allow the operator to plan for thetrajectory of a surgical instrument during the procedure. To facilitatethe planning, and as further shown in FIGS. 5 and 7, the system 200′,200″ may also include a plan interventional instrument 230′, 230″ thatis configured to permit the user to create a surgical interventionalplan and to store the surgical interventional plan on a memory of thecomputer 224′, 224″ of first image acquisition apparatus 206′, 206″(such as the MDCT or the C-arm angio fluoroscopy imager). In this case,it should be appreciated that the surgical interventional plan mayinclude a delineation of target tissue in the first holographic imagedataset and digital content representing a path of the surgicalinstrument to the target tissue.

With continued reference to just FIG. 5, the first image acquisitionapparatus 206′ or MDCT may further have an instrument position andorientation transmission module 232′ configured to transmit the surgicalinterventional plan from the plan interventional instrument 230′ to theaugmented reality system 204′.

It should be understood that the interventional plan of the presentdisclosure has digital (i.e., virtual) content representing a path of asurgical instrument to the target tissue or anatomical structures, whichmay be visually represented as surfaces (e.g., point cloud or vertices)by the augmented reality system 204′, 204″. The digital treatment orinterventional plan can be specified, for example, by selecting a 3Dpoint on the target tissue and drawing a line on the anatomical imagerepresenting a desired path avoiding critical structures.

In particular, the plan may be transmitted to the augmented realitysystem 204′, 204″ in the form of an Augmented/Extended Reality (AR)headset, for example, also referred to as a head mounted display (HIVID)or a suitable device with similar capabilities. Prior to transmission,i.e., at the time of installation, the pre-registration may be performedso that the plan is transformed to coordinates of the HMD. This enablesthe virtual representation of the planned trajectory, for example, as aholographic light ray (HLR), and target tissue to be augmented on to thereal patient as a holographic content in the HMD (world) coordinates. Inturn, this allows the operator to align a physical instrument such as amedical probe for treatment and/or diagnosis. Live imaging acquired fromthe imaging system can then be used to re-register, as described furtherhereinbelow, the treatment plan including digital structures derivedfrom a 3D dataset and instrument trajectories for improved alignment tothe target.

2. Pre-registration of 3D Planning Data into HMD coordinates

Prior to the first use of the augmented reality system 204′, 204″ or HMDin combination with the system 200′, 200″, the HMD may pre-registered byintegration with the system 200′, 200″ in accordance with the method100′, 100″ as further shown in FIGS. 6 and 8 relative to the specificexamples of the MDCT and C-arm angio fluoroscopy unit, respectively.This step is performed once at the time of the installation step 101′,101″, so that the interventional plan can be automatically transformedfrom imaging by the first image acquisition apparatus 206′, 206″ to HMDcoordinates leading to registration and augmented with and on to thepatient 201′, 201″. More generally, the pre-registration method is notlimited to use with pre- or intra-procedure treatment plans but can alsobe used to register a holographic navigation, i.e., tracked devicessystem with the system 200′, 200″.

It should be appreciated that the pre-registration method enables theHMD to transform treatment planning data from imaging system to HMD(world) coordinates without the need for the operator to performregistration steps during the procedure. Although the pre-registrationmethod 100′ is described in FIGS. 3 and 5-6 for the multi-detector rowCT (MDCT) imaging system, and the pre-registration method 100″ isdescribed in FIGS. 4A-4B and 7-8 for the C-arm angio fluoroscopy unit,it should also be understood that the methods are generally applicableto any 3D imaging system.

The digital treatment or surgical intervention plans can include theposition and trajectory (“planned trajectory”) of one or a plurality ofinterventional instruments, including points, linear and curved paths,as well as 3D coordinates representing structures in the operative siteas surfaces derived from 3D data set acquired with the imaging system.

A. Multi-detector row CT (MDCT) Embodiment

For the multi-detector row CT (MDCT) embodiment shown in FIGS. 3 and5-6, the first optical image target 220′ is placed on the imager 218′such as an imaging systems gantry of the MDCT with a line-of-sight tothe HMD. The first optical image target 220′ is used to determine a 3Drigid affine transformation from the imager (MDCT) spatial 3Dcoordinates to holographic coordinates for all future patients. Althoughnot depicted in FIGS. 3 and 5-6, it should be appreciated that anadditional image target can also be placed near the skin surface of thepatient 201′ so that upon detection of the HMD the HLR can terminate atthe skin surface representing the percutaneous access location.

In the case of the MDCT gantry, which remains stationary, as shown inFIGS. 5 and 6, the HMD can be calibrated with the CT gantry's coordinatesystem, for example, [x,y] origin at an aperture isocenter, z-axis alongpatient table. In this case, the trajectory planned on the CT consolemay then be transmitted to the HMD to augment the HLR in patientcoordinates, for example, as shown in FIG. 6. A guide for advancing theintervention instrument, as described herein, is thereby provided. TheHMD can also receive data to compensate for the movable CT tableposition reported on the gantry display.

In particular embodiments, as shown in FIGS. 3 and 6, the (x,y) originof CT coordinates ascribed to the CT reconstructed images can be relatedto the isocenter of the gantry or imager 218′. The MDCT gantry may nothave capability for tilting, which is common for CTs manufactured inrecent years. The first optical image target 220′ such as a Vuforia®target, commercially available from PTC Inc. in Needham, Massachusetts,or a similarly suitable image target, may be mounted in a precise,repeatable location of the MDCT gantry and may therefore be used todetermine the transformation from MDCT to HMD coordinates. Thiscalibration or integration is performed at the time of installation, andnot by the end-operator for each case, and then stored by the HMD.Advantageously, this enables the operator to use the HMD, and saidpre-registration transformation, without the need for the registrationsteps at the time of the image-guided intervention which would otherwiselead to inefficient workflow.

For use by the operator, a virtual needle or other interventionalinstrument and delineation of target tissue may be planned on thereconstructed MDCT images in the MDCT coordinates. These 3D coordinates,for example, start [x,y,z] and end point [x,y,z] of the instrument, andtarget structures represented as surfaces are transferred to theaugmented reality system 204′ or HMD. The MDCT table position may alsobe sent to the HMD at this time.

After planning the virtual instrument on the MDCT display, the operatormay then retrieve the HMD from the predetermined storage position 210′such as a storage pod, which is associated with a repeatable “home”position. The HMD then applies the MDCT-to-HMD pre-registrationtransformation to the virtual instrument and target structurecoordinates to project the holographic interventional instrument orneedle in the HMD (world) coordinates, so that it can be used as a guidefor the interventional procedure.

B. C-arm Angiography Fluoroscopy Unit (Cone beam CT or CBCT) Embodiment:

For the C-arm Angiography Fluoroscopy Unit embodiment shown in FIGS.4A-4B and 7-8, the first optical image target 220″ is placed on theimager 218″ such as a flat panel detector of the C-arm with aline-of-sight to the HMD. The first optical image target 220″ is used todetermine a transformation from the isocenter of the C-arm system (e.g.,originating at a center of the flat panel) to holographic coordinates ofthe HMD for all future patients. Although not depicted in FIGS. 4A-4Band 7-8, it should be appreciated that an additional image target canalso be placed near the skin surface of the patient 201″ so that upondetection of the HMD the HLR can terminate at the skin surfacerepresenting the percutaneous access location.

In a particular example, the first optical image target 220″ may be aVuforia® image target or similar optical target. The first optical imagetarget 220″ is mounted in a precise, repeatable location on the flatpanel detector of the C-arm used to determine the transformation from anisocenter of the C-arm system, i.e., originating at the center of theflat panel, to the augmented reality system 204″ or HMD coordinates. Aswith the MDCT embodiment shown in FIGS. 3 and 5-6, this calibration orintegration is performed at the time of installation, and not by theend-operator, and is stored by the HMD, for example, as shown in FIGS. 7and 8. This enables the operator to use the augmented reality system204″ in the form of the HMD and the first image acquisition apparatus206″ in the form of the C-arm with the pre-registration for theimage-guided intervention and without the need for additionalregistration steps which would otherwise lead to inefficient workflow.

In yet another example, for use by the operator, the optical imagetarget 220″ may be placed on the C-arm detector in a predetermined ordesignated location, e.g., where a calibration was performed, in orderto track its pose. A virtual needle or other interventional instrumentmay then be planned on the reconstructed Cone Beam CT images in thecoordinate system 214″ of C-arm Angiography Fluoroscopy Unit, asillustrated in FIG. 4A and shown in step 125″ in FIG. 8. These 3Dcoordinates, for example, start (x,y,z) and end points (x,y,z) of theinstrument, may then be sent or transferred to the C-arm system. TheC-arm gantry then aligns with the planned trajectory.

It should be appreciated that, although the detector's laser could beused to show the trajectory, a presence of the C-arm can in factinterfere with the access of interventionalist or operator to thepatient. Thus, the HMD may further use the first optical image target220″ to locate the C-arm pose. After calibration of the flat paneldetector into the HMD coordinates, the first optical image target 220″may be used to locate a holographic light ray (HLR) guide 234″ inalignment with the isocenter of C-arm gantry (shown in arbitrarylocation in FIGS. 4A and 4B). More specifically, the system may applythe memory stored C-arm-to-HMD transformation and the virtual instrumentcoordinates to project the holographic needle and/or instrument andtarget structures in HMD (world) coordinates so that the HMD maygenerate the holographic light ray (HLR) guide 234″, as shown in FIGS.4A and 4B. The C-arm may then be moved away, as shown in FIG. 4B, withthe HLR guide 234″ remaining. Advantageously, the ability to move theC-arm but retain the HLR guide 234″ provides sufficient clearance forimplementation of the procedure.

With respect to FIGS. 4A and 4B, an image tracker target is attached andcalibrated to the C-arm flat panel detector. After calibration of theflat panel detector into the HMD coordinates, the image tracker targetis used to locate a holographic light ray (violet line) in alignmentwith the isocenter of C-arm gantry (shown in arbitrary location). Theoperator can then provide a voice command such as “place” that keeps theHLR stationary after the C-arm is moved away to provide sufficient spacethe interventionalist or operator to position the physicalinterventional instrument.

III. Re-Registration

FIGS. 9-10 illustrate a method 300 and a system 400 of image-guidedintervention involving re-registration, as described further herein.Like or related steps or structure from FIGS. 1-2 are shown in FIGS.9-10 with the same reference number but in a 300- or 400-series, forpurpose of clarity.

In particular embodiments, the method 300 and the system 400 involvingre-registration may be used in conjunction with the method 100 and thesystem 200 involving pre-registration, as described hereinabove.However, it should be appreciated that the re-registration method 300and the system 400 may also be used independently from thepre-registration techniques of the present disclosure, as desired.

The method 300 includes a step 302 of providing the system 400, as shownin FIG. 10, for performing the image-guided intervention on a patient401 including an installation 402 of an augmented reality system 404 ata first image acquisition apparatus 406. As non-limiting examples, thefirst image acquisition apparatus 406 may be one of a multidetector rowcomputerized tomography (MDCT) imager and a C-arm angio fluoroscopyimager. One of ordinary skill in the art may also select other suitabletypes of imagers for the first image acquisition apparatus 406 withinthe scope of the present disclosure.

The augmented reality system 404 is in communication with a computer 408having a processor 409 and a memory 411. The memory 411 may includenon-transitory processor-executable instructions directing the augmentedreality system 404 to depict the first holographic image datasetadjacent to the patient 401. The computer 408 may be configured toreceive the surgical interventional plan from the first imageacquisition apparatus 406.

The augmented reality system 404 may be initially disposed at apredetermined storage position 410 relative to the first imageacquisition apparatus 406. The augmented reality system 404 has anaugmented reality system coordinate system 412, and the first imageacquisition apparatus 406 has a first image acquisition apparatuscoordinate system 414. The augmented reality system coordinate system412 is different from the first image acquisition apparatus coordinatesystem 414.

In certain examples, the computer 408 may also be configured totransform the first image acquisition apparatus coordinate system 414 tothe augmented reality system coordinate system 412 for the firstholographic image dataset according to the pre-registrationtransformation technique, as described hereinabove.

With respect to the re-registration technique described herein, thecomputer 408 may have an initial transformation of the first imageacquisition apparatus coordinate system 414 into the augmented realitysystem coordinate system 412 stored in the memory of the computer 408.For example, the initial transformation may be the pre-registrationtransformation as described hereinabove; however, one skilled in the artshould appreciate the initial transformation may be provided by othermeans and is not necessarily the pre-registration transformation.

The first image acquisition apparatus 406 further has a patient table416 for receiving the patient 401 and an imager 418 for acquiring afirst holographic image dataset from the patient 401 while on thepatient table 416.

With further reference to FIG. 10, the system 400 further includes asecond image acquisition apparatus 436 configured to acquire a real-timeholographic image of the patient 401 during an image-guidedintervention. The second image acquisition apparatus 436 has an imager438 and a second image acquisition apparatus coordinate system 440. As anonlimiting example, the second image acquisition apparatus 436 may bean ultrasound transducer and the imager 438 may be a probe of theultrasound transducer. One of ordinary skill in the art may also selectother suitable imaging systems for the second image acquisitionapparatus 436 within the scope of the present disclosure.

With renewed reference to FIG. 9, the method 300 further include a step319 of acquiring with one of the first image acquisition apparatus 406and the second image acquisition apparatus 436 a real-time holographicimage dataset of the patient 401 during the image-guided intervention.The real-time holographic image dataset may define a two-dimensionalplanar sector, for example. The real-time holographic image dataset maythen be compared in a step 330 with either a position of the patient 401(e.g., by a simple visual comparison performed by the operator of theaugmented reality system 404) or the first holographic image dataset(e.g., by an automated or semi-automated process performed by a computerof the system 200). In a step 332, the first holographic image datasetmay then be aligned with the real-time holographic image dataset fromthe patient to provide a re-registration transformation. There-registration transformation may then be stored, in a step 334, in thememory 411 of the computer 408 of the augmented reality system 404.

In certain examples, the initial transformation prior to there-registration technique maybe be the pre-registration transformation,as described hereinabove. In such case, the initial transformation maybe determined by the method 100 (shown in FIG. 1) including i) a step104 of placing a first optical image target 420 at the predeterminedlocation on the imager 418 of the first image acquisition apparatus 406,ii) a step 106 of acquiring first optical image target coordinates fromthe first optical image target 420 with the augmented reality system404, and iii) a step 108 of determining the initial transformation asthe pre-registration transformation of the first image acquisitionapparatus coordinate system 412 into the augmented reality systemcoordinate system 414 using the first optical image target coordinates.

With renewed reference to FIGS. 9 and 10, the second image acquisitionapparatus 436 configured to acquire the real-time holographic imagedataset of the patient 401 during the image-guided intervention may beused in the step 319 of acquiring the real-time holographic imagedataset of the patient 401 with the second image acquisition apparatus436. Advantageously, where the first image acquisition apparatus 406 isone of a multidetector row computerized tomography (MDCT) imager and aC-arm angio fluoroscopy imager, and the second image acquisitionapparatus is an ultrasound transducer, the employment of the ultrasoundtransducer for acquiring the real-time holographic image dataset mayhelp minimize radiation exposure to the patient 401.

Where the second image acquisition apparatus 436 is the same as thefirst image acquisition apparatus 406, such as the MDCT or C-arm angiofluoroscopy imager, it should be appreciated that the system may furtherinclude a module with an algorithm used to automatically segment theMDCT to form a surface, plane, or sector in real-time. A skilled artisanmay select a suitable algorithm for automatically segmenting the imagesfor this purpose, as desired.

Example Re-Registration Techniques:

With reference to FIGS. 9-12, further exemplary embodiments associatedwith the re-registration techniques described hereinabove are shown andexplained.

1. Re-registration to Account for Movement of Target Tissue:

The re-registration method 300, as described hereinabove, can be used tocompensate the initial registration, such as the pre-registrationaccording to the method 100, for motion of the patient 401. The motionmay be either gross or physiological motion, for example. There-registration method 300 uses live imaging from the same or anotherimaging system to update and improve the registration of the treatmentplan to the augmented reality system 404 or HMD coordinates prior toperformance of the procedure.

Although live images have a limited field-of-view (FOV), and 3Dtomographic imaging does not reflect breathing motion, their combinationaccording to the present disclosure may be used to improve the accuracyof initial registration, and this correction for inaccuracy is generallyreferred to herein as “re-registration.” Live imaging may be used toupdate the transformation of the target tissue (and other structuresderived from the static 3D data set) and the associated plannedtrajectory to increase the accuracy when implementing the treatmentplan.

Spatial registration methods, including the pre-registration method 100described hereinabove, will transform 3D spatial coordinate datasetssuch as planned interventional trajectories and anatomical targets basedon pre- or intra-procedural 3D data set. For example, such methods mayuse rigid-body affine matrices to transform (i.e., rotate and translate)the 3D coordinate data into the HMD (or world) coordinates. In contrast,the live imaging such as CT, fluoroscopy, or sonography techniques canreflect motion and deformation but can have the limited field-of-view(e.g., single plane or projection as shown in FIG. 11).

The re-registration method 300 is not limited to use with thepre-registration method or, more generally, is not limited to use withan interventional treatment plan. The re-registration method 300 can beused to update any initial or preliminary registration (such as manualor fiducial marker methods) between live imaging and static 3D imagingdata including inter- and intra-modality combinations thereof, forexample, between live sonography (two spatial dimensions and time), andsegmented 3D (three spatial dimensions) multi-detector row CT data.

As shown in FIG. 11, the camera geometry associated with the liveimaging system imager or transducer of the second image acquisitionapparatus 436 may also be modeled. For example, the modeling may includea plane or sector for intra-modality care of CT fluoroscopy (as well asthe inter-modality case of sonography). Likewise, the modeling mayinclude a projection frustum for the case of fluoroscopy. Further, themodeling may include a camera frustum for the case of endoscopy.

2. Intra-procedure Initial Registration or Pre-registration:

In reference to FIGS. 9 and 10, it should be appreciated that theinitial registration of the planned trajectory and delineated and/orsegmented anatomical structures to the augmented reality system 404 orHMD (world coordinates) can be performed using the rigidpre-registration method 100, described hereinabove. For the initialregistration, the camera geometry is also registered to the HMDcoordinates, either intrinsically, e.g., as in the intramodality case ofCT fluoroscopy, or extrinsically, e.g., as in the case of sonography.The treatment plan and holographic live images are then both located inHMD coordinates and stored on the augmented reality system 404.

3. Refinements of the Initial Registration of Key Structures 3D Datasetto Live Imaging:

For the case of a planar or sector field-of-view, such as 2D sonographyand MDCT fluoroscopy, intersection points of the segmented target tissueand the live camera field-of-view may be determined, as described morespecifically in reference to the examples shown FIGS. 13-20 furtherherein.

Generally, however, in re-registration one or more contours may be shownon the live camera image of the augmented reality system 404, asillustrated in FIG. 12. In FIG. 12, the intersection points are shown indashed lines with key structures such as target tissue or other criticalstructures shown in solid lines.

In accordance with the step 330 of FIG. 9, and associated FIGS. 10-12,the comparison may be performed between the intersection contours andthe live imaging of the structure, for example, as shown at the left inFIG. 12. In a semi-automated method, the comparison can be performedvisually by the operator of the augmented reality system 404. This canbe performed at a breath hold or at suspension of breathing duringventilation, for example, where the movement of the patient 401 may beminimized. The transformation of the key structures in the 3D data setis then updated so that the contours align or are congruent with thelive imaged structures, for example, as further shown at the right inFIG. 12.

For the case of the projective field-of-view of the camera (e.g.,fluoroscopy or endoscopy), the comparison under the step 330 of themethod 300 may also be made either a) in the common projection of thekey structures, or b) at the intersection of the field of viewdelineation lines, or c) at the projection lines of key landmarks, asdesired by the operator.

The re-registration transformation may then be determined manually byadjusting the 3D representation (e.g., holographic) of key structures inaccordance with the step 332 of the method 300. The adjusting may beperformed using hand gestures or handles on the anatomical hologram, asnon-limiting examples. The transformation can also be adjustedautomatically or semi-automatically. One automated method may include anincremental adjustment of the 3D translation (Tx, Ty, Tz) and rotations(Rx, Ry, Rz) by the computer, and evaluating a registration figure ofmerit (e.g., the Boolean intersection of the imaged edge-contour and theintersectional contour of the imaged at each increment). The incrementaladjustment may be selected on each iteration that results in animprovement in the figure of merit until an optimization criterion ismet. The intersection points and contours are then updated on theaugmented reality system 404.

After the re-registration is performed, the contours of the live imagedkey structure and the 3D key structure of the treatment plan may besubstantially congruent, which is indicative of the planned trajectorybeing accurately aligned. Other suitable means for either manually,semi-manually, or automatically adjusting the first holographic imagedataset to align with the real-time holographic image dataset from thepatient to provide the re-registration transformation may also beemployed, as desired.

More generally, and as set forth hereinabove, the re-registration method300 can be used for other preliminary registration methods (i.e.,re-registration is not limited to use with pre-registration). Forexample, the initial registration of the treatment plan (e.g., probetrajectory and rigid anatomical holograms) to HMD display coordinate maybe based on a fiducial marker method instead of the pre-registrationmethod as described hereinabove.

4. Intersectional Contours:

As set forth hereinabove, it should be appreciated that there-registration method 300 may employ the use of intersectionalcontours, which are obtained by comparing the first holographic imagedataset from the patient with the initial transformation to a real-timeholographic image dataset of the patient during the image-guidedintervention. It should be appreciated, however, that the intersectionalcontour technique may also have applications other than just there-registration method. Certain examples for generating theintersectional contours for either re-registration or otherapplications, for example, as shown in FIGS. 13-20, are describedfurther hereinbelow.

FIG. 13 illustrates a method 500 of image-guided intervention involvingthe use of intersectional contours for re-registration, as describedfurther herein. Like or related steps from FIGS. 1, 6, 8, and 9 areshown in FIG. 13 with the same number but in a 500-series, for purposeof clarity.

In particular, the method 500 of image-guided intervention for thepatient 201, 401 may include both a step 518 of acquiring with the firstimage acquisition apparatus 206, 406 the first holographic image datasetof the patient 201, 401, and a step 519 of acquiring with the secondimage acquisition apparatus 436 the second holographic image dataset ofthe patient 201, 401 during the image-guided intervention. In particularembodiments, the second holographic image dataset may be a real-timeholographic image, and in more particular embodiments, the firstholographic image dataset is a three-dimensional image and the secondholographic image dataset is a two-dimensional planar sector image, bothof which can be shown to the operator via the augmented reality system204, 404.

In a step 530, the first holographic image dataset is then compared withthe second holographic image dataset. For example, the comparison mayinclude an overlaying of the first holographic image dataset with thesecond holographic image dataset, for example, as shown in FIGS. 12 and14.

In a step 531, the intersectional contour of the first holographic imagedataset on the second holographic image dataset may be determined. Forexample, the intersectional contour may be defined by intersectingpoints of the three-dimensional image of the first holographic imagedataset and the two-dimensional planar sector of the real-timeholographic image. It should be appreciated that a series of theintersecting points may define a boundary line of the intersectionalcontour. In certain examples, the first image acquisition apparatus 206,406 is one of the multidetector row computerized tomography (MDCT)imager and the C-arm angio fluoroscopy imager, and the second imageacquisition apparatus 436 is the ultrasound transducer.

More particularly, the step 531 of determining the intersectionalcontour of the first holographic image dataset on the second holographicimage dataset may include calculating the portion of the firstholographic image dataset that is not the intersectional contour fromthe view of the operator. In this case, the intersectional contour maybe a vertex or mesh element retained in a stereoscopic projection, wherea distance value is less than a threshold value (k) that is preset orpredetermined by the operator. It should be appreciated that thethreshold value (k) may define a dimension of the intersectional contourrelative to the two-dimensional planar sector.

The threshold value (k), and its employment in defining the dimension ofthe intersectional contour, is also described further in the algorithmshown in FIG. 14. The threshold value (k) is selected to be large enoughfor the intersectional contour to be visually seen with the augmentedreality system 204, 404, without being so large that it undesirablyobstructs the view of the patient and/or the target tissue by theoperator. The threshold value (k) can also be a negative (-) or apositive (+) value, or both, as the dimension of the intersectionalcontour may extend on either side of the two-dimensional planar sector,as desired. It should be appreciated that, in certain examples, thethreshold value (k), has a value of between 0.1 mm and about 2 mm, andmost particularly about 1 mm. However, one of ordinary skill in the artmay select other suitable values for the threshold value (k) within thescope of the present disclosure.

The step 531 further includes a step of removing, by the computer 208,408, a portion of the first holographic image dataset that is not theintersectional contour from a view of the operator of an augmentedreality system 204, 404. In this example, only the intersectionalcontour of the first holographic image dataset is shown overlaid on thesecond holographic image dataset to the operator. It should beappreciated that the threshold value (k) may be predetermined or may beadjusted by the operator in order to show a desired dimension of theintersectional contour. The desired dimension may be one that does notadversely obstruct a view of the patient by the operator. Other suitablemeans for determining the intersectional contour from the comparison ofthe first holographic image dataset with the second holographic imagedataset may also be employed, as desired.

Following the step 531, the method 500 may include a further step 532 ofaligning the first holographic image dataset with the intersectionalcontour, whereby the re-registration transformation may be performed asset forth in the method 300 described hereinabove. Likewise, there-registration transformation performed using the intersectionalcontour may be stored for further use in the image-guided intervention,as also set forth in the re-registration method 300.

5. Example of Image Guided Intervention with Intersectional Contours:

An exemplary embodiment of an intervention 600 by an operator 601 forthe patient 201, 401 and utilizing the intersectional contour technologyis further shown in FIGS. 15-20. In FIG. 15, the intervention 600 isshown including a three-dimensional holographic image 602 of the patient201 compared with a real-time two-dimensional sector image 604 of thepatient 201, 401. As non-limiting examples, the three-dimensionalholographic image 602 of the patient 201, 401 may be obtained from thefirst holographic image dataset, which in turn is acquired from thefirst image acquisition apparatus 206, 406 such as the multidetector rowcomputerized tomography (MDCT) imager or the C-arm angio fluoroscopyimager. Similarly, the real-time two-dimensional sector image 604 of thepatient 201, 401 may be obtained from the second holographic imagedataset, which in turn is acquired from the second image acquisitionapparatus 436 such as the ultrasound transducer.

As further shown in FIG. 15, where an intersectional contour line 606 isinitially generated, the intersectional contour line 606 may beinitially offset from the real-time image (shown right in FIG. 15). Inorder to adjust for this inaccuracy, which may be caused by patientmovement, as one non-limiting example, a location of thethree-dimensional holographic image 602 of the patient 201, 401 asviewed through the augmented reality system may be adjusted as describedherein.

In one example, the real-time two-dimensional sector image 604 of thepatient 201, 401 may be further provided with controls 608 with whichthe operator 601 may interact to manually cause movement of thethree-dimensional holographic image 602 of the patient 201, 401 relativeto the real-time two-dimensional sector image 604. The movement may becaused by the operator until the three-dimensional holographic image 602of the patient 201, 401 is substantially aligned or congruent with theintersectional contour line 606 (shown left in FIG. 15). As describedfurther hereinabove, either semi-automatic or fully automaticadjustments may also be employed in lieu of the manual adjustmentmethod, as desired.

More particularly, as shown in FIG. 16, the operator 601 may begin theintervention 600 by placing at least one optical image target 610adjacent the patient 201, 401, or in this case a portion of the patient201, 401, such as a breast 612 of the patient 201, 401, on which theintervention 600 is to be performed. The operator 601 may further wearthe augmented reality system 204, 404 in the form of the head mounteddevice of the present disclosure, and view the patient 201, 401 throughthe augmented reality system 204, 404.

In particular, FIG. 16 shows the view of the operator 601 during anultrasound scanning of the breast 612 of the patient with an ultrasoundprobe 614 being pressed against the breast 612 of the patient 201, 401.Another image target (not shown) may be used on the ultrasound probe614, which can be used to determine a transformation from the trackedultrasound probe 614 to the three-dimensional holographic image 602 ofthe patient 201, 401.

The augmented reality system 204, 404 also permits for a view of thephysical interventional device 616 and an associated holographic lightray 618 to be used for the intervention 600. The at least one opticalimage target 610 is further shown in an offset position from the breast612 in FIG. 16, and permitting for the employment of the augmentedreality system with the intervention 600, for example, due toimplementation of the pre-registration and/or re-registration methods asdescribed hereinabove.

For the intervention 600, and as shown in FIG. 17, the operator 601 thenmay select with the augmented reality system 204, 404 to view thethree-dimensional holographic image 602 of the patient. Thethree-dimensional holographic image 602 shows target tissue for theintervention 600, as well as surrounding vasculature, for example.

Further in support of the intervention 600, and as shown in FIG. 18, theoperator 601 may then select with the augmented reality system 204, 404to view the real-time two-dimensional sector image 604 of the patient.In this case, the real-time two-dimensional sector image 604 from theultrasound scan may be overlaid with the three-dimensional holographicimage 602 of the patient from the CT scan. In this case, both of thethree-dimensional holographic and real-time two-dimensional sectorimages 602, 604 may also be provided as translucent so as to permit theoperator 601 to also view the breast 612 of the patient for theintervention 600.

It should be appreciated that the operator 601 may then, as shown inFIG. 19, cause the augmented reality system 204, 404 to generate theintersectional contour lines 606 of the target tissue on the real-timetwo-dimensional sector image 604, i.e., the holographic ultrasound planevisualized through the augmented reality system 204, 404. As shown inFIG. 20, the intersectional contour lines 606 will also change dependingon the movement of the ultrasound probe 614, such that a firmer press ofthe ultrasound probe 614 against the breast 612 of the patient may showthe intersectional contour lines 606 at a plane through the targettissue that is different from a plane through the target tissue that maybe shown with a softer press of the ultrasound probe 614 against thebreast 612 of the patient.

Advantageously, the method 100 and the associated system 200 of thepresent disclosure enables pre-registration for 3D holographic guidanceand navigation, which leads to less steps for the operator and moreeffective workflow. It is believed that the efficiency of the method 100and the system 200 will facilitate increased adoption of the method 100and the system 200. Likewise, the method 300 and the system 400 of thepresent disclosure, which enables intra-procedural re-registration of 3Dholograms derived from 3D tomographic data with live imaging, includingre-registration through use of intersectional contour techniques, isbelieved to improve accuracy and safety of the probe placement duringinterventional procedures.

Example embodiments are provided so that this disclosure will bethorough, and will fully convey the scope to those who are skilled inthe art. Numerous specific details are set forth such as examples ofspecific components, devices, and methods, to provide a thoroughunderstanding of embodiments of the present disclosure. It will beapparent to those skilled in the art that specific details need not beemployed, that example embodiments may be embodied in many differentforms, and that neither should be construed to limit the scope of thedisclosure. In some example embodiments, well-known processes,well-known device structures, and well-known technologies are notdescribed in detail. Equivalent changes, modifications and variations ofsome embodiments, materials, compositions and methods can be made withinthe scope of the present technology, with substantially similar results.

What is claimed is:
 1. A method of image-guided intervention for apatient, the method comprising steps of: providing an installation of anaugmented reality system at a first image acquisition apparatus, theaugmented reality system in communication with a computer with aprocessor and a memory, and the augmented reality system having anaugmented reality system coordinate system and the first imageacquisition apparatus having a first image acquisition apparatuscoordinate system, the first image acquisition apparatus further havinga patient table for receiving the patient and an imager for acquiring afirst holographic image dataset from the patient while on the patienttable, and the installation further including steps of: placing a firstoptical image target at a predetermined location on the imager;acquiring a first optical image target coordinates from the firstoptical image target with the augmented reality system; determining apre-registration transformation of the first image acquisition apparatuscoordinate system into the augmented reality system coordinate systemusing the first optical image target coordinates; and storing thepre-registration transformation in the memory of the computer, whereby apre-registration of the first image acquisition apparatus with theaugmented reality system is performed; and following the installation ofthe augmented reality system and the pre-registration of the first imageacquisition apparatus with the augmented reality system, applying thepre-registration transformation to transform the first holographic imagedataset from the first image acquisition apparatus coordinate system tothe augmented reality system coordinate system.
 2. The method of claim1, wherein the step of acquiring the first optical image targetcoordinates from the first optical image target with the augmentedreality system includes moving the augmented reality system so that thefirst optical image target is in a field of view of the augmentedreality system.
 3. The method of claim 2, wherein the step of applyingthe pre-registration transformation occurs during the image-guidedintervention, the image-guided intervention further including steps of:placing a second image target on the patient while on the patient table;acquiring a second optical image target coordinates from the secondimage target with the augmented reality system; acquiring a firstholographic image dataset of a portion of the patient on the patienttable; creating a surgical interventional plan using the first imageacquisition apparatus, the surgical interventional plan including adelineation of target tissue in the first holographic image dataset, thesurgical interventional plan provided in the first image acquisitionapparatus coordinate system; transmitting the surgical interventionalplan from the first image acquisition apparatus to the augmented realitysystem; transforming the surgical interventional plan from a first imageacquisition apparatus coordinate system of the first image acquisitionapparatus into the augmented reality system coordinate system using thepre-registration transformation from the installation; and generating aholographic light ray on the augmented reality system to show anoperator a trajectory for an instrument according to the surgicalinterventional plan.
 4. The method of claim 3, wherein the patient tableis translatable, and further comprising at least one of: determining andregistering a position of the patient table with the augmented realitysystem; placing an additional image target at the patient table,acquiring an additional image target dataset from the additional imagetarget with the augmented reality system, and determining a position ofthe patient table by the augmented reality system based on theadditional image target dataset; and transmitting a position of thepatient table from a patient table sensor on the patient table to theaugmented reality system.
 5. The method of claim 3, further comprising are-registration to compensate the pre-registration transformation forinaccuracy.
 6. The method of claim 5, wherein the re-registrationincludes steps of: acquiring with one of the first image acquisitionapparatus and a second image acquisition apparatus a real-timeholographic image dataset of the patient during the image-guidedintervention; comparing the real-time holographic image dataset with thepatient or the first holographic image dataset; and adjusting the firstholographic image dataset to align with the real-time holographic imagedataset from the patient.
 7. The method of claim 6, wherein the firstholographic image dataset defines a three-dimensional image volume andthe real-time holographic image dataset defines a two-dimensional planarsector.
 8. The method of claim 7, wherein the portion of the firstholographic image dataset is an intersectional contour defined byintersection points of the three-dimensional image volume of the firstholographic image dataset and the two-dimensional planar sector of thereal-time holographic image dataset.
 9. The method of claim 8, whereinthe first image acquisition apparatus is one of a multidetector rowcomputerized tomography (MDCT) imager and a C-arm angio fluoroscopyimager, and the second image acquisition apparatus is an ultrasoundtransducer.
 10. A system of image-guided intervention for a patient,comprising: an installation of an augmented reality system at a firstimage acquisition apparatus, the augmented reality system incommunication with a computer with a processor and a memory, andaugmented reality system having an augmented reality system coordinatesystem and the first image acquisition apparatus having a first imageacquisition apparatus coordinate system, the first image acquisitionapparatus further having a patient table for receiving the patient andan imager for acquiring a first holographic image dataset from thepatient while on the patient table, wherein a pre-registrationtransformation is stored in the memory of the computer, thepre-registration transformation determined by steps of: placing a firstoptical image target at a predetermined location on the imager;acquiring a first optical image target coordinates from the firstoptical image target with the augmented reality system; and determininga pre-registration transformation of the first image acquisitionapparatus coordinate system into the augmented reality system coordinatesystem using the first optical image target coordinates.
 11. A method ofimage-guided intervention for a patient, the method comprising steps of:providing an installation of an augmented reality system at a firstimage acquisition apparatus, the augmented reality system incommunication with a computer with a processor and a memory, and theaugmented reality system having an augmented reality system coordinatesystem and the first image acquisition apparatus having a first imageacquisition apparatus coordinate system, an initial transformation ofthe first image acquisition apparatus coordinate system into theaugmented reality system coordinate system stored in the memory of thecomputer, the first image acquisition apparatus further having a patienttable for receiving a patient and an imager for acquiring a firstholographic image dataset from the patient while on the patient table,acquiring with one of the first image acquisition apparatus and a secondimage acquisition apparatus a real-time holographic image dataset of thepatient during the image-guided intervention; comparing the real-timeholographic image dataset with the patient or the first holographicimage dataset; adjusting the first holographic image dataset to alignwith the real-time holographic image dataset from the patient to providea re-registration transformation; and storing the re-registrationtransformation in the memory of the computer.
 12. The method of claim11, wherein the initial transformation is a pre-registrationtransformation determined by the steps of: placing a first optical imagetarget at a predetermined location on the imager; acquiring a firstoptical image target coordinates from the first optical image targetwith the augmented reality system; and determining a pre-registrationtransformation of the first image acquisition apparatus coordinatesystem into the augmented reality system coordinate system using thefirst optical image target coordinates.
 13. The method of claim 11,further comprising steps of providing the second image acquisitionapparatus configured to acquire the real-time holographic image datasetof the patient during the image-guided intervention, and acquiring thereal-time holographic image dataset of the patient with the second imageacquisition apparatus.
 14. The method of claim 13, wherein the firstholographic image dataset is a three-dimensional image and the real-timeholographic image dataset is a two-dimensional planar sector.
 15. Themethod of claim 14, wherein a portion of the first holographic imagedataset is an intersectional contour, the intersectional contour definedby intersection points of the three-dimensional image of the firstholographic image dataset and the two-dimensional planar sector of thereal-time holographic image dataset.
 16. The method of claim 15, whereinthe first image acquisition apparatus is one of a multidetector rowcomputerized tomography (MDCT) imager and a C-arm angio fluoroscopyimager, and the second image acquisition apparatus is an ultrasoundtransducer.
 17. A system of image-guided intervention for a patient,comprising: an installation of an augmented reality system at a firstimage acquisition apparatus, the augmented reality system incommunication with a computer with a processor and a memory, and theaugmented reality system having an augmented reality system coordinatesystem and the first image acquisition apparatus having a first imageacquisition apparatus coordinate system, an initial transformation ofthe first image acquisition apparatus coordinate system into theaugmented reality system coordinate system stored in the memory of thecomputer, the first image acquisition apparatus further having a patienttable for receiving the patient and an imager for acquiring a firstholographic image dataset from the patient while on the patient table,wherein a re-registration transformation is stored in the memory of thecomputer, the re-registration transformation determined by steps of:acquiring with one of the first image acquisition apparatus and a secondimage aquisition apparatus a real-time holographic image dataset of thepatient during the image-guided intervention; comparing the real-timeholographic image dataset with the patient or the first holographicimage dataset; and adjusting the first holographic image dataset toalign with the real-time holographic image dataset from the patient toprovide the re-registration transformation.
 18. A method of image-guidedintervention for a patient, the method comprising steps of: acquiringwith a first image acquisition apparatus a first holographic imagedataset of the patient; acquiring with a second image acquisitionapparatus a second holographic image dataset of the patient during theimage-guided intervention; comparing the first holographic image datasetwith the second holographic image dataset by overlaying the firstholographic image dataset with the second holographic image dataset;determining an intersectional contour of the first holographic imagedataset on the second holographic image dataset; and removing a portionof the first holographic image dataset that is not the intersectionalcontour from a view of an operator of an augmented reality system,whereby only the intersectional contour of the first holographic imagedataset is shown overlaid on the second holographic image dataset. 19.The method of claim 18, wherein the second holographic image dataset isa real-time holographic image.
 20. The method of claim 19, wherein thefirst holographic image dataset is a three-dimensional image and thesecond holographic image dataset is a two-dimensional planar sector. 21.The method of claim 20, wherein the intersectional contour is defined byintersection points of the three-dimensional image of the firstholographic image dataset and the two-dimensional planar sector of thereal-time holographic image.
 22. The method of claim 21, wherein thestep of determining the intersectional contour of the first holographicimage dataset on the second holographic image dataset includesdetermining the portion of the first holographic image dataset that isnot the intersectional contour from the view of the operator, theintersectional contour being a vertex or mesh element retained in astereoscopic projection where a distance value is less than a thresholdvalue (λ) preset by the operator, the threshold value defining adimension of the intersectional contour relative to the two-dimensionalplanar sector.
 23. The method of claim 22, further including a step ofadjusting the threshold value (λ) by the operator in order to show theintersectional contour without adversely obstructing a view of thepatient by the operator.
 24. The method of claim 23, wherein the firstimage acquisition apparatus is one of a multidetector row computerizedtomography (MDCT) imager and a C-arm angio fluoroscopy imager, and thesecond image acquisition apparatus is an ultrasound transducer.
 25. Themethod of claim 18, further comprising steps of: providing aninstallation of an augmented reality system at the first imageacquisition apparatus, the augmented reality system in communicationwith a computer with a processor and a memory, and the augmented realitysystem having an augmented reality system coordinate system and thefirst image acquisition apparatus having a first image acquisitionapparatus coordinate system, an initial transformation of the firstimage acquisition apparatus coordinate system into the augmented realitysystem coordinate system stored in the memory of the computer, the firstimage acquisition apparatus further having a patient table for receivingthe patient and an imager for acquiring the first holographic imagedataset from the patient while on the patient table; and providing thesecond image acquisition apparatus including an imager for acquiring thesecond holographic image dataset from the patient.
 26. A system ofimage-guided intervention for a patient, comprising: an installation ofan augmented reality system at a first image acquisition apparatus, theaugmented reality system in communication with a computer with aprocessor and a memory, and the augmented reality system having anaugmented reality system coordinate system and the first imageacquisition apparatus having a first image acquisition apparatuscoordinate system, an initial transformation of the first imageacquisition apparatus coordinate system into the augmented realitysystem coordinate system stored in the memory of the computer, the firstimage acquisition apparatus further having a patient table for receivingthe patient and an imager for acquiring a first holographic imagedataset from the patient while on the patient table; and a second imageacquisition apparatus including an imager for acquiring a secondholographic image dataset from the patient, wherein an intersectionalcontour is shown on the augmented reality system, the intersectionalcontour determined by steps of: acquiring with the first imageacquisition apparatus the first holographic image dataset of thepatient; acquiring with the second image acquisition apparatus thesecond holographic image dataset of the patient during the image-guidedintervention; comparing the first holographic image dataset with thesecond holographic image dataset by overlaying the first holographicimage dataset with the second holographic image dataset; determining theintersectional contour of the first holographic image dataset on thesecond holographic image dataset; and removing a portion of the firstholographic image dataset that is not the intersectional contour from aview of an operator of the augmented reality system, whereby only theintersectional contour of the first holographic image dataset is shownoverlaid on the second holographic image dataset.